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ABSTRACT 

This report to the United States Senate deals with 
the systems and procedures used by various federal agencies to award 
research funds to universities and the resulting distribution of 
those funds. More specifically, it is intended to review previous 
studies of the relationship between the award process and 
distribution of federal research funds, and to describa certain award 
procedures of the National Institutes of Health (NIH) and the 
National Science Foundation (NSF). The docunent is divided into ten 
sections. The first two sections provide general background on peer 
review. Sections three through six describe the NSF and NIH award 
processes, focusing on the role of external peer reviewers and 
internal agency officials. The remaining four sections address the 
issues of fairness and accountability in the awards processes, 
including what past studies have shown about peer reviews. The 
appendices include a selected bibliography of studies of peer review, 
listings of resources and criteria used by NSF and NIH when selecting 
peer review committees, and a breakout of NSF and NIH support given 
to under-represented groups, institutions, women, and minorities. 
(TU) 
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B-221714 
March 26 r 1987 

The Honorable Mark 0. Hatfield 
Ranking Minority Member^ Committee 

on Appropriations 
United States Senate 

Dear Senator Hatfields 

This fact sheet responds to your December 17^ 1985^ request 
that we examine the systems and procedures used by various 
federal agencies to award research funds to universities and 
the resulting distribution of those funds. In discussions 
with your office^ we agreed to (1) determine the 
distribution of federal research funds to universities and 
colleges by institution^ state ^ agency^ and field of 
science^ (2) analyze the extent to which patterns of 
distribution are accounted for by historical trends^ direct 
congressional action^ field of science, demographic and 
socioeconornic factors, and the use and distribution of peer 
reviewers, (3) review previous studies of the relationship 
between the award process and distribution of federal 
research funds, and (4) describe certain award procedures at 
the National Institutes of Health (NIH) and the National 
Science Foundation (NSF) . 

A companion report (GAO/RCED-87-^67BR ) covers points one and 
two and this fact sheet addresses points three and four* 
Our purpose is to describe the formal policies and 
procedures at NIH and NSF for making research awards* Due 
to congressional concern about the fairness and 
effectiveness of that part of the award process known as 
"peer review," we particularly focus on the role assigned to 
external scientists who review the research applications 
(i.e., the peer reviewers). In doing this work, we did not 
evaluate the agencies' award policies and procedures, nor 
did we review how well agency officials implement them* 

This fact sheet is divided into 10 sections. The first two 
sections provide general background on peer review; sections 
three through six describe the NSF and NIH awards processes, 
focusing on the role of external peer reviewers and internal 
agency officials; and the remaining sections address the 
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issues of fairness and accountability in the awards 
processes^ including what past studies have shown about peer 
review* 

We obtained information on NSF and NIH review processes by 
examining agency and congressional hearing documents^ 
interviewing federal agency officials ranging from program 
officers to senior management officials ir the Washington^ 
D^C^^ area^ -and reviewing past studies on the award decision 
processes* We discussed a draft of this fact sheet with 
responsible NIH and NSF officials who generally agreed with 
its contents* Their technical comments have been^ 
incorporated where appropriate* 

We are sending copies of this fact sheet to relevant 
committees of the Congress^ the Director of NIH, the 
Director of NSFr and to other interested parties upon 
request* If you have additional questions or if we can be 
of further assistance in this matter^ please contact me at 
(202) 275^1000* 

Major contributors are listed in Appendix X* 
Sincerely yours ^ 

'Barah P. Prazier 
Associate Direetor 
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SiCTION 1 



Wiat is the "pe^ review* systin used by fe-ideral agencies to 
award research grants ana how did it begin? 

Many federai agencies fuMing researoh at universities ask 
tor the adyice expert ptofeiaionals , or — 'paers" of 

mvesta^gators on grant applications, to hel^ them decide which 
proposed proaects to fund. Ms part of thS agenc=ies • decision- 
making process as known ag a peir review system. 

Pmmr review in science bpn at least 300 years ago, when 
the meiTEbtra of toe Royal society in England adopted a referee 
system -to evaluate papers Cot publication. Currently/peer 
review is used principally itj three wayei Cirst/ to assist in 
choosing which research projects should be Cunded ; second? to 
select ^hich research papers should be publiXshid ±n a journal or 
sc^Snnf^^-'J^'^i^^' third, to judge the =inaings in'an area of 

made on rL^^t^^^ ^nCom^tm, such as wl^in a lecision must be 
Sff ^2 P°^®"t^^^ regulation^ or priorities for agency programs. 
This rei5ort focuses on the fint. ^ f y 

In the unit&d States, theestatalishmeht . of the National 
Advisory Cancer Council in 1 931 marked the b-efinning of federal 
use of peer review. The Coundl was authori aid to review 
applications for research funding and to cer tify approval to the 
Surgeon Gineral f^r proJeGts that had the po-^ential of 
^f^Si!''?*f-^ con-tributing to knowledge abou^ canaer. The office 
of Naval Research (ONR) also mi peer review to assist in its 
universjLty research funding inthe late 1 940 'i. ONR grants 
managers would informally seeKadvice by experts in the field 

forresearch funding. ONR -s approach 
was later adopted and formali^ei by the Natic=nal Science ° -"^ 

commfn?i°;^''®^^i ^^^'^ P^^^ '''''' P'^o^es^ elicits written 
comments from external peers, 

TccS^""^^^' Within the l^ational Institutes of Health (nih) 
and NSP, external peer review Is relied upon pradominantly in 
awarding research projects.! other agencies vary considerably in 

lAn advisory committee appointd to assess MSP's peer review 
system, in a repoirt; to the Ditactor, NSF ( Filial Repor t, nsf 
Advisory Committed o n Merit Review. Septemb er 25, 1 986) 
ricommended that mSF adopt thft torm "merit r^viaw" to describe 
Its award selecti»n process. Writ review, a^ccording to the 
irector -s Statement in the r^prt, takes int.o account the 
evolution of the p^«er review prociss, wherebr" the changing scale 
?^.^^r"iftion o± science have Increased th .e importance^ol 
«ators other than scientiSic rarlt on the avr^ard decision 

?£ ?hls; fSors!'^"" ' °' ^""'"'''^ ^^^^^ r discussion of some 
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how they assess the merit of proposals, with practices that may 
irprfgram stSff! ^^""'"^^ external peers, internal experL 
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SECTION 2 
Why is peer review controversial? 

Peer review is an inexact, subjective process in which 
scientists judge which researchers and which proposals are most 
likely to yield the most fruitful results. Common criticisms of 
the present federal use of peer review are; 

— institutions concentrated in the last, the Midwest, and 
the West historically have received the major share of 
federal research funds. (During the mid 1940 's, 
geographic concentration of university research was one 
of the major points of congressional hearings on a bill 
that was introduced to establish NSP.)2 

— Peer review . . is an ^old boys' system' where program 
managers rely on trusted friends in the academic 
community to review their proposals. These friends 
recommend their friends as reviewers. ... it is an 
incestuous 'buddy system' that frequently stifles new 
ideas and scientific breakthroughs . "3 Sixty-three 
percent of the approximately 4,100 scientists who 
responded to a 1986 Sigma Xi (a national honorary 
scientific society) survey agreed with the statement that 
"Procurement procedure for grants to do governmentally 
sponsored research depends on 'who you know'. Many 
requests seem to be funded primarily because the 
researchers are already known to and 'supported by the 
granting organizations . "4 



- Peer review and the present grants system in general are 



See our companion report. University Fundings Patterns of 

Distribution of Federal Resea rc h Funds to Universities , 

(GA0/RCED-87-67BR) , for analysis of present federal research 
funding patterns. This report states that although distribution 
of total federal research funds to institutions is concentrated 
in a few states and institutions, related factors, such as fields 
of science and population, influence the patterns of distribution. 

3Representative John Conlan, National Science Foundation Peer 
Reviews Special Oversight HearingH Subcommittee on Science, 
Researcn, and Technology, committee on Science and Technology, 
U.S. House of Representatives, 94th Congress, 1st session, page 5. 

*See A New Agenda for Science (preliminary report). New Haven, 
Connecticut s Sigma Xi, She Scientific Research Society, 1986, 
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too conservative. 5 Additionally, peer review is not 
appropriate for selecting proposals that are either" 
feseafch*6 ^""o^ative or incorporate interdisciplinary 

— The paperwork and time spent by reviewers detracts too 
much from the national research effort. Established 
scientists undertaking worthy research should receive 
awards for longer periods with less need for lengthy 
documentation in requests for initial awards and for 
renewals./ 

Despite these views, peer review has been continually and 
overwhelmingly endorsed as the best method of assuring that the 
best research is funded. Polls of researchers have yielded 
strong support for the use of external peers to assist federal 
agencies in determining which proposals should be funded. For 
example, 86 percent of the respondents in the above-mentioned 
Sigma Xi survey favored the use of peer review in research 
program,!. Additionally, in studies over the past 30 years, 
nn^Si^^ distinguished scientists, presidential commissions, and 
numerous witnesses at congressional hearings have endorsed the 
use of peer review.8 (see sec. 10 for a summary of studiesfsee 
app. I for a selected list of studies.) see 



5see Rustum Roy, "Funding Science i The Real Defects of Peer 
ilMI.'od)? fi!fr?|^i.'° jgi^nca. Technology, and Hn..n 



6see A. L. Porter and F.A. Rossini, "Peer Review of 
^"j^mlum^'n^'" iSil^. Technology, and Human 

7see Rustum Roy, "Funding Sciences The Real Defects of Peer 
Valuls/^g tS)f TirfMs! Technology, and Hnm.n 



^Iw^nT^ recent endorsement of the peer review system is by the 
NSF Advisory Committee on Merit Review, in its 1986 reportf the 

to'hf ""f,:, P*«^ system "has blen found 

nLS® remarkably effective and flexible, adapting to the changing 
needs of science and engineering research, and incorporating^ 
improvements with respect to openness, accountability, equity, 
and impact on the research environment." See Final Repolt NSF 
Advisory committee on Merit Review, nsf, Septe mber 2b!^19ii 
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How do NSF and NIH select rasearal» projects for award to 
aniversitles and colleges? 

This section describes NSP*s and NIH's processes for 
awarding research grants to indivL dual university scientists, 9 

NSF PROJECT SELECTION PROCESS 



ectives in selecting projects 

NSF's project selection proce as for awarding individual 
project grants is designed to sele ct projects that are perceived 
to be of the highest scientific an--d technical merit. It is also 
designed to encourage wider parties ipation from under-^represented 
groups and institutions and to su^^ort innovative ^ high-risk 
research. ^ 0 



Project selection process 

During fiscal year 1 985 , NSP ^ook final action on 24^403 



competitively reviewed propoials 
almost twice as many^ 15^504 , were 
withdrawn by applicants prior to a 
awards were for individual project 
$75^000 in total costs. 



Of these^ 7^968 were awarded^ 
declined- and 931 were 
decision. 11 Nearly all of the 
grants; the average award was 



The system that NSP has eitab^ished to decide which 
individual grant proposals to awarrf is shown in figure 3.1. A 
brief description of the process f follows. 



^This fact sheet focuses on the seZB^ection processes of the 
dominant funding mechanism used by both NSP and NIH^ namely^ the 
project grant awarded to thi individual university investigator. 
For a discussion of federal fundin^g mechanisms and the relative 
emphasis federal agencies place on them^ see our report 
University Funding; Federal Fundic^g Mechanisms in Support of 
University Research (GA0/RCED--86-53 # February 19861 . 



lO^he terms " under representid grou^js and institutions" and 
"innovative^ high-risk research" a^e defined in section 7. 

^Testimony delivered by Dr, Mary C^lutter^ NSF Senior Science 
Advisor^ on April 9^ 1986^ before t— he Task Force on Science 
Policy^ Committee on Science and Technology^ House of 
Representatives . 



It 



Figure 3.1: Key Sieps In IhB NSF pfoltel SBltcHon Proeeis 




Sources GAO, based on agency documents and other publiclv 
available sources. 

Most grant proposals received by NSF are unsolicited, when 
received, proposals are assigned to officers within program 
divisions according to their field of science or enginllrfng. 
After receiving the proposal, the program officer gives a 
?rni^n^"*-^ assessment, selects external peer reviewers, usually 
from universities (sometimes from industry), and asks them to 

lirovlrflf'JatJnl ja"^"^^'"^"^ ^ "i^^^-^ 

In receiving external advice from peer scientists, the 
program officer generally uses one of three procedures i ad hoc 

nfni/f!^®'"' P^"-^ ^ combination of mall ind 

thfL^r In ad hoc mail review, the program officer sends 

the proposal to several people (sometimes as many as ten), along 
thiv ^"^t^^ctions, and reviewing forms that contain the criteria 
nrnL! in evaluating the proposal. On the average, each 

proposal IS actually reviewed by five to six people. in laAels 
only, the program officer typically assembles 8 L 12 reviewira 

^ critique the proposals; this approach is mostly used 
when a sizable number of standard proposals are submitted for 
special programs, such as the Presidential Young Investigator 

NSP®LrS?^°L^ a discussion of who the peer reviewers are at 

NSF and NIH and how they are chosen. 
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Awards. In a combination of ad hoc mail review and panels^ the 
program officer sends the proposals out to both mail reviewers 
and to panelists* The panelists then meet in Washington to 
evaluate the proposals^ also taking into account the comments of 
the ad hoc mail reviewers. 

After receiving external peer evaluations, the program 
officer evaluates the proposals, taking into consideration both 
external peer comments and ratings and his or her own judgment of 
scientific merit. He or she also considers other factors, such 
as under represented researchers, less research-intensive 
institutions, and innovative, high-^risk research, before 
recommending whether an award should be made. 

The program officer's recommendation is then reviewed at one 
or possibly two higher levels (section and/or division) before an 
official decision is made. At these higher review levels, the 
program officer's recommendation can be questioned, returned for 
more documentation, or overturned (according to NSP officials, 
the latter is rare). For awards of at least $1.5 million 
annually, or of a total commitment of $6 million, the National 
Science Board, the body charged by statute with establishing NSF 
policies, must review the staff decision before an award can be 
made * 

NIH PROJECT SELECTION PROCESS 



Objectives in selecting project s 

NIH selects research projects with the intent of funding 
biomedical research of the highest scientific and technical 
merit. NIH also selects projects with the intent of supporting 
research relevant both to its own research program needs and to 
larger societal health concerns. 

Project selection process 

In fiscal year 1986, NIH reviewed 19,119 competitive 
research project grant applications. Of these, 13,682 were new 
and 5,105 were competitive renewals (i.e., applications seeking 
renewal contingent upon another peer review), and 332 were 
competitive supplements. About a quarter of the new applications 
and a little over half of the renewals were funded. Most of 
these applications were for unsolicited individual research 
project grants; the average award was $149,532 in total costs. 13 



I^Excluded from the above counts are withdrawals that occur 
before the advisory council meets, applications subsequently 
amended, resubmitted, and reviewed during fiscal year 1986, and 
all other applications that are not defined as research projects. 
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The mechanism NIH uses to review all grant applications is 
called the "dual review" system* Dual review refers to 
scientific peer review of project applicatiorie by "initial review 
groups" and then a second review by statutorily mandated 
institute advisory councils or boards. 

The NIH project selection process is shown in figure 3*2 and 
a description of the key steps in this process follows. 



PIgurt 3,2: Key iteps in the NfH Project 
ieffction Procais 




Source: GAO, based on agency documents and other publicly 
available sources* 



)lication assignment 

Applications sent to NIH are first received at the Division 
of Research Grants {DRG)14 that reports to the Director of NIH 
and is located outside of the individual program institutes that 
make the award* Referral officers within DRG assign 
applications to one of approximately 90 DRG peer review 
committees, called initial review groups or study sections, that 
are organized by specific fields of research (e*g*r molecular 
biology, nutrition, biochemistry). They also assign the 
application to the relevant funding institute, bureau, or 
division within NIH* 

Initial review for scientific merit 

Most competing project applications are first peer-reviewed 
for scientific and technical merit within DRG study_sections that 
meet on or near the NIH campus three times a year. 1 5 During 
these meetings, peer reviewers discuss each application using 



14drg serves as a staff resource for central receipt, assignment, 
and referral to the institutes of all applications for Public 
Health Service research and training support* It also provides 
the initial scientific review for most NIH research grants. 



ISlndividual NIH institutes also manage peer review of some 
applications in initial review groups similar to those manaqed 
by DRG* 
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evaluation criteria that NIH developed, and then vote whether the 
application has sufficient mar it to warrant funding. Individual 
peer reviewers then assign each application voted worthy of 
funding a priority rating from which NIH staff compute an overall 
priority score. This score becomes part of each application's 
"summary statement" that is sent to the next level for review. 

Institute and advisory counci l 
review for program relevance ~ 

Each institute has its own statutorily mandated advisory 
council that reviews the summary statements, and where necessary^ 
the applications, before recommending approval for funding. The 
advisory councils are made up of scientists and lay persons who 
reGOmmend action on most applications by voting "en bloc" 
concurrence with the recommandations of the initial review groups 
or study sections. According to NIH officials, less than 10 
percent of the applications are individually singled out for 
special discussion (e.g., high program relevancyi special health- 
related need; inappropriate peer reviewi foreign applications ; 
ethical issues) at the advisory council meetings. 

Institute staff develop funding strategy 

Once the advisory council has acted on the initial review 
groups' or study sections' recommendations, or taken special 
action, institute staff rank the approved proposals based on 
their priority scores (or percentile values derived from them) to 
initiate a funding strategy based on available institute funds. 
Since scientific merit is not the only criterion used in making 
funding decisions, lower-rated proposals that are identified as 
having particular value, such as high relevance to NIH program 
needs, may be funded in place of higher-rated proposals. 
According to NIH officials, about 1 to 2 percent of the 
applications are funded in this way. 
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SECTION 4 



Wio are the peer reviewers and how are they chosen? 

While most of the pear reviewers that NSP and NIH ask to 
serve are active research scientists employed by universities as 
faculty members^ some others are scientists in research 
institutes^ government, or private industry, m will briefly 
describe, for NSP and for NIH, who they are, the resources used 
to recruit them, and the criteria used to select them. 

NSF PEER REVIEWERS 

According to April 1986 testimony by NSP*s senior science 
advisor, in fiscal year 1985 NSP asked 59,725 persons to serve as 
reviewers and panelists, reviewing approximately 24,000 
proposals. They came from a wide range of institutions, 
including large and small academic institutions # industry, 
government, and foreign research and educational organisations. 
A list of NSP reviewers and advisory panel members is available 
annually. 

How NSF peer reviewers are recruited 

NSP policy guidelines caution against too frequent use of 
any one reviewer, urging program officers to "constantly seek 
referrals of new persons for future use as reviewers 1° 
According to NSP testimony and what NSP officials told us, 
program officers look for qualified and knowledgeable reviewers 
from all backgrounds and use a variety of resources for 
recruiting them, ranging from lists of reviewers suggested by the 
applicants or current peer reviewers themselves^ to contacts made 
by NSF staff at professional meetings. AppendiK II contains a 
list of sr^cific resources NSF officials have told us they use, 

NSF peer review selection criteria 

NSP has written criteria for selecting peer reviewers who 
review by mail and/or serve on panels ^^'^ The most important 
criterion is that the reviewer have appropriate knowledge of the 
science. Other written criteria include consideration for 
geographic and institutional balance and for the use of women and 
minorities. For panels, written criteria include consideration 
for age distribution and for balanced membership similar to that 



io NSP Proposal and Award Manual No. 10 , Section 122^4(e). 

17Mail reviewers are chosen by NSF program officers. Panel 
reviewers are nominated by NSP program officers and approved 
higher level NSF officials. 



for mail review* Appendix ill summarizes specific NSP criteria 
for selecting peer reviewers, 

NIH PEER REVIEWERS 

NIH has approximately 2,700 active external peer reviewers 
as part of 155 advisory committees that meet three times a year 
at or near the NIH campus to review competitive project 
applications . 1 8 About 2,300 of these reviewers are members of 
initial review groups that assess the applications' scientific 
merit, and another 400 are on institute advisory councils and 
program advisory committees that assess their relevance to 
program and societal goals. The membership of these groups is 
published yearly in NIH Public Advisory Groups , 

Most members who serve on initial review groups are active 
researchers at colleges and universities, over 50 percent of whom 
are full professors* Scientists who serve on an institute's 
advisory council or board are more senior and are chosen for 
their leadership in a discipline allied to the particular 
institute's mission. A large proportion are full professors, or 
university or medical school administrators, such as deans and 
vice presidents, and have served in other advisory roles for NIH 
in the past, particularly on the initial review groups. 

Lay members of the revisory councils are private citizens 
who have demonstrated a special concern for a particular health 
problem and are usually active in related foundation or 
association work such as the American Cancer Society. Advisory 
committees also have a small percentage of government-^employed 
scientists and private industry researchers. 

Recruitment of initial peer reviewers 

Membership in an initial peer review group is generally 
limited to 4 years. Since there are about 2,300 peer reviewers 
serving on initial peer review committees at any one time, 500-- 
600 reviewers must be replaced each year* New members are 
nominated by the executive secretary of each committee and 
approved by the Director, NIH. Appendix IV provides details on 
the sources of names drawn upon. 

Selection criteria for initial peer reviewers 

The primary criterion for serving on an initial review group 
is demonstrated competence and achievement as an independent 
investigator in a scientific or clinical discipline or research 



1 8The 2,700 reviewers do not include those used on ad hoc panels 
or through mail review. 
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specialty. This may be measured by quality of research 
accomplished, publications, and other honors and activities. 
Other criteria include mature judgment and objectivity as well as 
the ability to participate adequately in the advisory group 
structure. 

In addition, there are a number of constraints that 
influence the composition of the initial review groups. For 
example, there are specific numerical goals for state and 
regional representation and for representation of women and 
minorities. Written waiver requests are submitted for exceptions 
to these as well as other requirements . i 9 Appendix V lists the 
NIH peer review selection criteria as well as constraints. 

Recruitment of advisory council members 

Whereas initial review group members are appointed by the 
Director, NIH, advisory council members, generally a third of 
whom are lay members and the remainder, physicians and 
scientists, are appointed by the Secretary, Department of Health 
and Human Services (HHS).20 m addition to nominations from NIH, 
the Secretary receives potential names from Congress, special 
interest groups, other organizational components within HHS, and 
from the general public. Within this framework, the Secretary 
makes the final decisions concerning appointments and issues the 
invitation letter to the prospective member. 

SelectiorL criteria for council members 



Advisory council or board members are chosen for their 
leadership in a discipline, their wide-ranging interests in a 
field, and their understanding of societal health needs. 

In factoring geography and under represented groups into the 
selection of advisory council members, NIH officials told us that 
NIH adheres to a policy of putting at least one female and one 
minority member on each advisory council nomination slate. Tn 
addition, an institute director told us that selecting two people 
from the same state is avoided if at all possible. ' " 



l^in fiscal year 1986, approximately ten percent of the total 
number of NIH nominations contained waiver requests. 

20The National Cancer Advisory Board is appointed by the President 
not the Secretary, HHS. 
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SECTION 5 



What, are the peer reviewers asked to do? 

In both NSF and NIH^ peer reviewers are asked to evaluate 
the sclantific merit of project applicationF, using p'rescribed 
agency evaluation criteria and to provide this evaluation in 
written form and orally (if panels are used) to agenGy staff for 
their consideration in selecting projects for funding. Peer 
reviewers are also asked by both agencies to review project 
applications in the context of broader program and societal 
issues , 

THE ROLE OF NSF PEER REVIEWERS 

NSF offiqials ask peer reviewers to provide a written 
evaluation of the scientific and technical merit of a proposal 
using four NSF-wide proposal evaluation criteria: 

research performance competence^ that is^ the technical 
soundness 1^ the capability of the investigator^ and the 
adequacy of institutional resources available for the 
work; 

the intrinsic merit of the research^ that is^ the extent 
to which the proposal is expected to lead to new 
discoveries or fundamental advances in its field or 
across fields of research; 

~ the utility or relevance of the research^ that is^ the 
extent to which the work could contribute to an extrinsic 
goal such as a new technology; and 

" the effect on the infrastructure of science and 

engineering / that is^ what the work will contribute to 
the nation's research^ education^ and human resource 
base . 

NSF officials we spoke with said that peer reviewers 
concentrate mostly on the first two criteria^ commenting on the 
scientific method in the proposal and the capability of the 
investigator to carry it out. The third criterion is used more 
for applied research proposals ^ especially in the area of 
engineering^ and the fourth criterion^ according to some 
officials we spoke with^ is not addressed very often by the 
external peer reviewers. The fourth criterion does^ however^ 
permit the program officer to evaluate and Foundation officials 
to award proposals because of their potential for improving the 
nation's research base (including women and minorities)* 

Peer reviewers participating in NSF*s ad hoc mail procedure 
are asked to provide an overall rating of the proposal as well as 




detailed written eomments on a specific evaluation form that 
lists and describes the above four criteria. If participating in 
panels, peer reviewers orally discuss the merits of a group of 
proposals, with designated reviewers leading discussions and 
providing detailed written comments for certain proposals and a 
panel summary . 

Once the peer reviewers have evaluated the proposals, their 
formal role ends. For those serving on panels, NSF provides 
travel eKpenses and usually a $100 per day honorarium. Ad hoc 
mail reviewers are not compensated/ ~ ~^ 

THE ROLE OF NIH PEER REVIEWERS 

AS part of its dual review system, NIH has established two 
sequential Revels of peer review panels with distinct roles. The 
first peer review panel, the initial review group, provides a 
scientific assessment of grant applications i decides by majority 
vote whether applications are worthy of funding; and may make 
budget and duration revisions when recommending applications for 
approval. 

The second peer review group, the advisory council, assesses 
the guaiity of the initial scientific review; evaluates program 
priorities and relevancei and recommends to institute staff 
applications for award. Whether or not they are app/roved or 
disapproved by the initial review group, all applications above 
$50,000 direct costs each year go through the second level of 
review. 

In order to evaluate the scientific and technical merit of 
each application, peer reviewers who conduct the initial review 
are given the following review criteria that are established by 
Public Health Service regulations. These criteria are 

(1) the significance and originality of the proposal from a 
scientific and technical point of view, 

(2) the adequacy of the methodology to carry out the 
research, 

(3) the qualification and experience of the principal 
investigator and staff, 

(4) reasonable availability of resources (equipment, etc.)r 
and 

(5) reasonableness of the proposed budget and duration of 
the project. 

(6) other factors, such as human subjects, animal welfare, 
and biohazards* 
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During panel meetings^ peer reviewers^ using the above criteria^ 
lead a discussion of upplications for which they prepared written 
critiques. Following discussion^ peer reviewers introduce a 
motion for each of the applications to approve^ disapprove , or 
defer y and then the members vote openly. Some applications may 
be recommended for approval by the peer reviewers with a revised 
budget and duration. All peer reviewers then individually and 
privately assign each application voted worthy of approval a 
priority rating from one to five^ one being the best. Each 
application's rating is multiplied by 100 and averaged to yield a 
priority score as a 3^digit number that may range from 100 to 
500. Once the Initial review group has completed this process^ 
its formal role is over. 

The advisory council members have the legislatively mandated 
task of recommending applications for award. In doing so^ they 
may or may not concur with the recommendations of the initial 
review groups. Specifically^ advisory counGil members are asked 
by the institutes to assess the quality of the initial scientific 
review as well as the application's relevance to institute 
research program goals and broader societal health-related 
matters • 

In advance of each meeting, council members are sent 
briefing books that contain the initial review group's summary 
statements. The institute staff may bring to the attention of 
the council particular applications for discussion of 
considerations such as high program relevance or promising new 
areas of research. In addition, council members may raise for 
discussion particular applications where policy issues beyond 
scientific merit need to be considered. As noted in section 3, 
most applications are not singled out for discussion, and the 
Council votes concurrence with the initial review group "en 
bloc." 

Once the council members have recommended for or against all 
applications, their formal role in the project selection process 
ends. The council members, as well as members of other NIH 
advisory groups, receive travel expenses, per diem, and a $100 
per day honorarium for their service on the panels. 
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SECTION 6 



What Is the role of agency officials In the project selection 
process? 

While peer reviewers advise NSP and NIH agency staff on the 
scientific merit of each research project application, agency 
staff, in turn, are responsible for (1) ensuring that the 
applications have been assessed in the most objective and 
balanced manner possible, (2) ensuring that broade - agency 
policies and procedures are followed, and (3) recommending to 
agency officials at higher levels which applications should be 
funded , 

The key figency official assigned to these responsibilities 
withm the NSF project selection process is the program officer, 
while m the NIH system these responsibilities are shared by 
health scientist administrators, some of whom are executive 
secretaries of review groups and some of whom are institute 
program managers. We will briefly describe their roles in the 
award selection process. 

THE NSF PROGRAM OFFICER 

There are two kinds of program officers s the "rotators," 
who are on leave from universities to NSF for i- or 2-year 
periods, and the regular career program officers, many of whom 
started as rotators and then stayed on. There are about 300 
program officers, of whom about 100 are rotators. Because NSF 
rotators serve short terms as NSF program officers, it is 
important to describe how they are selected. 

According to officials we talked with, NSF widely advertises 
Its rotator openings and also calls people who have served on NSF 
review panels. Each division is expocted to keep a list of 
possible rotators. The primary criterion for choosing rotators, 
according to officials, is their expertise in the scientific 
field. Officials told us that the selection of rotators also 
follows the equal employment opportunity guidelines that are 
applicable to all outside hires. 

What the program officer or j-otator does 

The role of the NSF program officer or rotator includes 
ensuring an appropriate peer review of the application- 
negotiating budgets with the applicant; recommending to the 
Foundation which applicatiors should be funded* and, when 
recommending, considering the effect of the proposed research on 
the nation's research infrastructure. 
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Selects appropriate peer reviewers 



Program officers are responsible for ensuring that 
appropriate^ qualified peer reviewers are selected and an 
acceptable number of peer reviewers address each of the four 
evaluation criteria. For example^ NSP proposal guidelines state 
that the Poundation prefers that all proposals be reviewed by 
four to eight reviewers. When fewer than three are used^ the 
program officer submits a special justification, 21 

Recommends proposals for funding 

The program officer is responsible for recommending to NSF 
which proposals should be funded. He or she bases this 
recommendation on (1) the comments of external peer reviewers 
whose advice was sought ^ (2) his or her own assessment of the 
scientific merit of the proposal^ and (3) the Foundation's wider 
policies governing proposal evaluation and selection. 

Although program officers are not bound by external peer 
comments, they must, when justifying their recommendations to the 
Foundation in the proposal jacket , include reasons for any 
significant deviation from them. For example, the program 
officer can deviate from external peer comments to support high- 
risk, creative research. All recommendations are reviewed at one 
or more higher levels before an actual funding decision can be 
made • 

Expected to consider 

the researcTi infrastructure 

Officials in the directorates with whom we spoke said that 
the program officer has the responsibility of making sure that 
the effect of the proposed research on the infrastructure of 
science and engineering is considered before making a formal 
recommendation. By infrastructure, NSF refers, for example^ to 
such factors as the quality of training of scientists and, in 
particular^ the influence of the type of research on the careers 
of graduate students; to the development of new interdisciplinary 
approaches to research; and to the support of special equipment 
needs, NSF officials with whom we have talked have said that, in 
general, this criterion acts as a "swing factor" in the program 
officer's recommending the proposal for funding. 

NIH EXECUTIVE SECRETARIES AND INSTITUTE STAFF 

The MIH executive secretaries manage the initial review of 
the application but they do not themselves assess scientific 
merit, NIH institute staff manage the second level of peer 



NSF Proposal and Award Manual, No* 10, section 122.3. 
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review, _ Identify applications of high or low program relevance 
and of innovative, high-risk research, and recommend to the 
institute director which proposals recommended for approval bv 
the advisory council should be funded. 

|xecutiva secretaries manage 
initial scientific review 

NIH executive secretaries are described as experienced 
health scientists, hired in career positions at a senior 
government level, most of whom have a research and administrative 
background. They are no longer active researchers, as are the 
rotators within NSF; nearly all of them, however, have earned 
doctorates . 

The primary responsibilities of the NIH executive 
secretaries are to nominate the peer reviewers for membership on 
panels; to officiate at the peer review panel meetings in order 
to ensure that NIH policies and procedures are followed; and, at 
the conclusion of the panel meetings, to write a summary 
statement for each application reflecting a balanced account of 
each application's panel peer review. 

After initial review group meetings are completed, and the 
executive secretaries have prepared summary statements for each 
application assigned to them and forwarded them to the institutes 
and after they have attended advisory council/board meetings to 
respond to questions, their direct responsibilities end 
However, if the council/board does not concur with the initial 
review group's evaluation of an application, it is returned to 
the NIH review organization for reconsideration. 

Institute staff recommend 
appl ica t To n f o r f undi hg 

The major responsibilities of the staff of each institute in 
application review and selection are to assist in nominating 
members for vacancies on advisory councils, to identify 
applications with special characteristics, such as program 
relevancy or innovative research, and to recommend to the 
institute director which of the applications recommended for 
award by the advisory council should be funded. After the 
council has met, institute staff prepare a ranked "pay list" of 
awards that in their judgment should be made. The institute 
director, who makes the final funding decision, can, based on 
these recommendations or other criteria, also fund a council- 
approved application that falls below the pay line if it has high 
program relevance or Is truly scientifically important. Appendix 
VI summarizes the responsibilities of both the NIH executive 
secretaries and the institute staff. 
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SECTION 7 



How do NSF and NIB consider underrepresentad groups of 
researchers r Institutions, and types of research when selecting 
research projects for award? 

Both NSF and NIH consider under represented groups of 
researchers^ institutions, and types of research when selecting 
research projects for award* Thay address some or all of these 
areas either as part of their regular research programs or 
through separately budgeted programs. NSP has set targets to 
particularly encourage .the support of women and minorities. 

In this section r we will describe the mechanisms each agency 
has established to address four areas of special concern 
regarding award selection. These areas are geographic 
distribution of awards^ underrepresented groups and institutions, 
innovative high-risk research, and young or new investigators* 

GEOGRAPHIC DISTRIBUTION OF AWARDS 

In addition to considering geographic distribution in the 
selection of peer reviewers (see section 4)^ NSF addresses 
geographic distribution of awards (1) by permitting the program 
officer to consider geographic balance when recommending project 
applications for funding and (2) through a centrally managed 
program that makes awards competitively to planning and 
management committees within states that have received the least 
federal research support over a period of time (this program is 
discussed in the section below). Although geography is 
considered in the selection of peer reviewers^ NIH does not 
consider geographic balance when selecting competing research 
project applications f nor does it have any separately budgeted 
programs that address geographic distribution of awards. 22 

NSF's Experimental Program to Stimulate 
Competitive Research (EPSCOR) 

The goal of NSF's Experimental Program to Stimulate 
Competitive Research (EPSCOR) is to increase the ability of 
scientists and engineers in participating states to compete 
successfully for federal research and development (R&D) funds by 
fostering long-term improvements in the research environments. 
NSF began EPSCOR in 1979 by giving planning awards to a state- 
wide group of scientists^ engineers^ and administrators in each 
of seven states that ranked lowest in R&D funding. The group had 



^^NIH does^ for technical reasons^ consider geography in some of 
its control and prevention initiatives, which may look at, for 
example, the relationship between groups of people in specific 
regions and certain incidences of cancer. 



9 months to assess tho state's science and technology base and to 
develop a 5-year research improvement plan. This pian could 
request up to $3 million. " 

Two rounds of planning avmrds have resulted in 5-year EPSCOR 
grants to groups in 12 states aiid 1 territory. m October 1980, 
NSP awarded such grants in Arkansas, Maine, Montana, South 
Carolina, and West Virginia, m August 1986, NSP awarded grants 
in Alabama, Kentucky, Nevada, North Dakota, Oklahoma, Puerto 
Rico, Vermont, and Wyoming. The 13 grants, ra-iging from $2.4 
million to $3 million each, totaled $36.9 million over a 5-vear 
grant period. 

NSP EPSCOR program officials told us that they are finding 
improvement in the ability of participants to compete nationally 
as a result of IPSCOR funding. For example. South Carolina 
received an EPSCOR award in 1980. In 1978, it had $900,000 in 
NSP competitive research program support (ranked 46th among all 
states). In 1986, it had $6.2 million and ranked 29th in NSP 
funding. Montana also increased its NOP awards markedly as a 
result of its EPSCOR program. 

UNDERREPRE5ENTED GROUPS AND INSTITUTIONS 

For purposes of this report, "underrepresented groups" are 
defined as ethnic minorities and women; "underrepresented 
institutions" are defined as predominantly undergraduate 
institutions, including small non-doctoral-granting liberal arts 
colleges and institutions granting a few doctoral-level degrees 
Both NSP and NIH have special mechanisms for addressing some or' 
all of these areas. 

NSP — Underrepresented researchers 
and institutions ~ 

NSP has a variety of mechanisms to encourage 
underrepresented groups and institutions. First, the general NSP 
evaluation criteria that apply to all proposals permit the 
program officers to consider underrepresented groups and 
institutions when recommending proposals for funding, and when 
selecting peer reviewers. Second, the Foundation puts out 
special announcements that are coordinated by a central group and 
that incorporate special program objectives regarding 
underrepresented groups and institutions into research program 
objectives. These programs have Foundation-set funding targets 
or goals. Third, research directorates have set aside funds to 
help m meeting these targets. 23 Fourth, NSP has several 



23piscal year 1986 NSF data show that, although NSP varied by 
individual tesearch directorate in meeting its targets for new 
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separately budgeted programs for women and minorities • Appendix 
VII shows a breakdown of these mechanisms and the funding 
estimated for each. 



NIH^-Under represented researchers 
and institutions 

NIH has separately budgeted programs to encourage 
participation by minorities^ minority institutions^ and small 
colleges in its biomedical research activities. In addition^ 
officials have estimated the extent to which women and minorities 
are supported through competing research project awards. 
Altogether^ NIH estimates that it awarded women investigators 
about 13 percent and minority investigators approximately 9 
percent of its competing research project obligations in fiscal 
year 1986. Appendix VIII contains a summary of the programs and 
levels of support. 

INNOVATIVE, HIGH-RISK RESEARCH 

A task force of the National Science Foundation Advisory 
Council identified three classes of "innovative high--risk" 
research: (1) research that challenges currently accepted 
scientific hypotheses and can lead to great advances in 
scientific knowledge (i.e., innovative), (2) research that 
transfers knowledge from one scientific field to another (i.e., 
interdisciplinary research), and (3) research that is at the edge 
of technical feasibility (i.e., risky projects that, if 
successful, would have high pay-^offs)^ We will describe 
mechanisms NSP and NIH have for addressing each of the above 
areas. 

NSF — Innovative, high-risk research 

NSP officials said they encourage innovative, high-risk 
research through their regular award review process. As 
discussed in section 5, one of NSP's four general criteria for 
evaluating all applications permits a consideration of the 
potential of the proposed research for improving the scientific 
and engii.eering enterprise, included in this goal are the 
stimulation of quality activities in important or underdeveloped 
fields and the use of interdisciplinary approaches to research in 
appropriate areas. Accocding to NSF officials, program officers 
are encouraged to seek out especially Innovative proposals and 
consider them for funding. One high NSP program official said 
she looks for how innovative proposals are treated when she 
reviews program officers' recommendations* 



minority researchers, women, and predominantly undergraduate 
institutions, overall, NSP was either close to meeting or 
exceeded its target in all three areas. 
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Some NSF programs have specific mechanisms to fund 
innovative research. The Engineering Directorate has an 
experimental program called EKpedited Awards for Novel Research, 
It allows a program officer to spend up to 5 percent of his or 
her budget on proposals that reflect new ideas. The grants are 
for 1 year^ and since there is no external peer review^ the 
program can respond quickly to new ideas. The Engineering 
Directorate also has a mechanism for funding "innovative and 
exceptional proposals" that are high risk, highly innovative , or 
interdisciplinary. In fiscal year 1985, eight proposals were 
funded for a total of $2 million. Finally, an NSF mechanism 
called Creativity Extension allows grants for projects that are 
judged to be truly creative to be continued for two additional 
years without having to be competitively renewed. 

NSF has other mechanisms specifically meant to aid 
interdisciplinary research. For example, NSF program directors 
in various scientific disciplines can directly stimulate 
interdisciplinary proposals by issuing special announcements in 
areas that combine technology with social issues, such as 
biotechnology or robotics and automated manufacturing. Also in 
the field of interdisciplinary research, the Engineering 
Directorate has an office of Engineering Infrastructure 
Development for coordinating the review of proposals that span 
the interests of several of its programs, 

NIH — Innovative, high-risk research 

NIH officials told us that they try to support innovative, 
high-^risk research through their regular competitive research 
project application program. In addition, NIH identified one 
activity called Small Grants that emphasizes one or more of the 
following purposes: pilot projects, testing of new techniques, 
and feasibility studies of high-risk research. The Small Grants 
activity also supports the following types of investigators: (1) 
recently trained or less-experienced investigators whose research 
careers were interrupted and are intended to be resumed, (2) 
investigators changing fields of research, (3) investigators at 
minority institutions or located in a largely nonresearch 
environment, and (4) established investigators needing quick 
support for a pilot project. In fiscal year 1985, this activity 
was provided agency funds of almost $4 million and made 191 
awards . 

NEW OR YOUNG INVESTIGATORS 

Both NSF and NIH have separately budgeted programs 
addressing new (first time in the system) or young investigators. 
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NSF""New investigatora 



NSP identified several programs for young or new 
investigators. Firsts several of the NSP directorates reported 
separate prograrns for new investigators such as the Engineering 
Initiation Awards that provide an opportunity for recently 
appointed assistant or associate professors to initiate academic 
engineering research. This mechanism is directed toward full-- 
time engineering faculty members who have had no prior 
substantial research supports In fiscal year 1984^ NSF funded $9 
million and made 227 awards for initiation grants in Engineering 
and Information Science, Second^ the NSF Presidential Young 
Investigators Award program provides initial support for 
promising young scientists and engineers^ In fiscal year 1984^ 
NSF funded about $24 million and made 200 awards under this 
program. (App. VII also describes programs for new women and new 
minority investigators.) 

NIH^ — New investigators 

NIH established in fiscal year 1986 a First Independent 
Research Support and Transition (FIRST) Award to provide a 
sufficient initial period of research support for newly 
independent biomedical investigators to develop their research 
capabilities and demonstrate the merit of their research ideas. 
These grants generally will be for 5 years. They replace the NIH 
New investigator Research Award that funded 812 awards in fiscal 
year 1984 at a level of $40 million. 



EKLC 



229 



SECTION 8 

How do NSP and MIH attenipt to maintain the openness of and 
monitor the fairness of their project selection systems? 

NSP and NIH have a number of specific procedures that 
attempt to maintain the openness of and monitor the fairness of 
the peer review system. We have identified in the previous three 
sections a number of these procedures that are built into both 
the selection of peer reviewers and the evaluation of proposals. 
This section will identify those procedures not previously 
discussed » 

MAINTAINING OPENNESS 

Maintaining openness refers to mechanisms that give the 
applicant feedback during or after the award selection process. 
Firsts both agencies make peer review evaluations available to 
the applicant. NSF automatically sends anonymous verbatim peer 
review comments to the applicants* NIH sends a summary statement 
to the investigator with the priority score displayed. Both 
agencies assert that sending reviewer comments to the principal 
investigators is a good training device to help the applicant in 
reassessing^ adjusting^ or improving his or her research program 
objectives. 

Second^ NSP and NIH have processes whereby an applicant can 
challenge the agency's decisions and ask that the application be 
reconsidered. in the NIH peer review appeals system^ the 
principal investigator first expresses his or her disagreement 
with the process and/or substance of the review to the institute 
directly responsible for the management of the application. If 
the principal investigator seriously disagrees with the response 
made by the institute^ the investigator and the applicant 
organization may jointly appeal to the Office of Extramural 
Research and Training^ a component of the Office of the Director^ 
NIH. The decision on an appeal is made at the highest level of 
NIH and it is final* 

An NSP applicant whose proposal has been declined may obtain 
an explanation from the responsible program officer. If this 
explanation does not satisfy him or her^ the applicant may also 
obtain reconsideration of the declination from several higher 
levels. (Reconsideration is an examination to determine if the 
decision on the proposal was arrived at properly in accordance 
with Foundation policies^) However^ neither NSP nor NIH can 
assure applicants that reconsideration will result in an award 
even if error is established in connection with the initial 
evaluation . 
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MONITORING FAIRNESS 



Pairness refers to mechanisms that have the purpose of 
assuring that an investigator's application received an objective 
or unbiased review* According to NSF officials, NSP monitors 
fairness when the program officer's award recommendations are 
reviewed by higher level officials. In addition, NSP "tests" the 
system post--award by both external and internal reviews. NIH, on 
the other hand, relies on the checks and balances of the dual 
review process and internal oversight, as needed, to validate the 
system's fairness, 

NSP--Monitoring fairness 

NSP has four procedures for monitoring fairness i (1) a pre- 
award division-level review of each proposal recommendation made 
by the program officer, 24 (2) post-award internal oversight, (3) 
post-award external oversight, and (4) statistical "fairness" 
evaluations by internal evaluation staff, 

Pirst, according to written NSP procedures, program 
officers' recommendations are routinely reviewed by the section 
head, if any, and by the division director to verify that the 
number and quality of the peer reviewers are adequatei that 
significant peer review comments contrary to the recommendation 
have been properly accounted for; and that the rationale for the 
recommendation is reasonable. Second, the staff of the 
Director's Office of Audit and Oversight routinely mines a 
small, random sample of awards for compliance with i policies 
and procedures and provides direct feedback to the program 
division. Third, every program is comprehensively reviewed every 
3 years by a small group of external peers familiar with the 
field* These "visiting committees" make publicly available 
formal reports to the NSP director about each program, 25 Fourth, 
NSP's program evaluation staff, which is independent of the 
program divisions, has run various statistical tests to see if 
the system has been "fair" to certain classes of applicants. 
(See sec. 10 for discussion*) 



24as discussed in section 7, NSF also has targets to encourage 
fairness and has recently created set-asides to ensure that these 
targets are met. 



25NSP has prescribed a set of questions that an external peer 
oversight committee must address when conducting its review and 
when reporting its findings. The questions cover such areas as 
the selection of an appropriate number of technically qualified 
reviewers; the reasonableness of the program officer's 
recommendation; and evidence of significant imbalance in various 
factors, including geographic distribution of principal investigators. 
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NI H — - Mo n i t o r 1 ng fairness 



An NIH policy document states that objectivity in the award 
decision procass is enhanced by 

— separating reviews for scientific and technical merit 
from reviews for program needs; 

— separating internal staff responsibility for scientific 
review from internal staff responsibility for program 
administration; 

reviewing a proposal's scientific merit on two levels ^ 
thereby providing a system of checks and balances (dual 
review) $ and 

" continuous quality control and oversight of study section 
reviews by experienced senior staff who visit study 
section meetings and review samples of summary 
statements » 

Several senior staff committees share responsibility for 
systematic, ongoing oversight of NIH's peer review system. For 
example, the Review Policy Committee reports to the Director of 
NIH, meeting monthly to share and discuss the progress of the 
initial review process. If NIH peer review problems surface 
within this oversight group, they recommend options for action to 
the NIH Director's office. 
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SECTION 9 



nhat are the key similarities and differences between the NSP and 
NIH systems for granting research awards to universities? 

This section summafizes the main similarities and 
differences between the NSP and NIH project selection systems - 
Appendix IX presents a chart comparing the award procedures of 
the two agencies* 

1^ The overall project selection process. Both NIH and NSF 
emphasise scientific merit when selecting among competing 
research project applications* Both agencies rely on the advice 
of external peer reviewers to determine such merit. NSF^ 
however^ permits its program officers^ many of whom are on 2-* 
year rotational assignments from their active research positions 
outside NSP^ to contribute to the evaluation of proposals for 
scientific merit, while NIH relies almost exclusively on its 
external peer reviewers. 

NSF uses a variety of peer review procedures, such as 
panels, ad hoc mail, and a combination of both. These procedures 
vary as much across NSP research directorates as they do within 
them. NIH, on the other hand, primarily uses the panel procedure 
with occasional use of ad hoc mail and a combination of panel and 
mail. Both agencies rely upon in-house staff to make final 
funding recommendations and decisions. 

2. Selection of peer reviewers. NSF and NIH written 
criteria for selecting peer reviewers emphasize their scientific 
expertise. They also have selection criteria that consider 
factors such as underrepresented groups and geographic balance. 
NIH, unlike NSP, requires that a specific percent of peer 
reviewers serving on study sections be women and minorities, and 
that they represent a certain regional and state balance. 

3. Role of peer reviewers in project selection. Both NSP 
and NIH ask peer reviewers to advise them on "the scientific and 
technical merit of project applications, NIH also asks peer 
reviewers to (1) recommend changes in budget and duration of 
approved applications, (2) numerically rank those proposals 
considered worthy of funding, (3) identify applications of high 
or low program relevance, and (4) recommend project applications 
for award, as statutorily mandated. 

4. Role of agency officials in project selection. Both NSP 
and NIH program officials make the final funding decisions. In 
this regard, NSP program officers formally recommend proposals 
for funding, and NIH institute staff construct pay lists based 
primarily on priority scores. NSP program officers, a third of 
which are "rotators" with recent research experience, evaluate 
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the scientific merit of proposals, whereas NIH officials rely 
external peers for such an aaaessmeht. 



5. Special infrastructure concerns. Both NSP and NlH have 
mechanisms to take into account underrepresented researchers and 
institutions as well as innovative, high-riaic research when 
making research awards. MSP has set targets for some of these 
infrastructure areas. 

6* Monitoring and oversight. In addition to agency review 
at higher levels, NSF relies on post-award review and oversight 
to assure the fairness and openness of its award selection 
system, while NIH relies on the checks and balance of the "dual 
review" system as well as regular quality control. 
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SECTION 10 



What have past studies aoncluded about the fairness and 
effactivenass o£ peer review in awarding research funds to 
univeraity scientists? 

Studies of past awards, surveys of scientists, and 
commission reports have endorsed the continued use of peer review 
and have not identified significant amounts of bias in the 
awards. However, a number of scientists believe there is some 
"cronyism." Only a few studies have attempted to evaluate the 
reliability and effectiveness of peer review in selecting 
research to be funded* 

Major studies of health or biomedical research by 
presidential commissions or panels of distinguished scientists 
have endorsed the continued use of peer review as a very 
effective way to select awards for university researchers. In 
addition, surveys over the last 10 years have similarly endorsed 
peer review. For example, of the approKimately 4,100 scientists 
responding to a 1986 Sigma Xi survey, 86 percent agreed that the 
government should use peer review to assure quality in its 
research programs. (See app. I for a selective list of these 
surveys and reports.) 

FAIRNESS 

Fairness addresses whether the system exhibits any bias 
toward the applicant related to such factors as age, sex, ethnic 
status, and the prestigfi or geographic location of his or her 
institution. By bias, we mean that peer reviewers or agency 
program staff, when faced with "two equally good proposals," 
would choose, for example, the one from a more well-known 
institution or from a more well-established researcher . 26 rphe 
perception that such a condition exists has prompted the 
accusation of the system being an "old boys ' network, " or 
exhibiting "cronyism." Although scientists overwhelmingly appear 
to endorse the continued use of peer review, there is a 
perception based on a number of surveys that some unfairness 
exists m the system. Analyses of NSF award decisions, however, 
do not provide conclusive statistical evidence to support these 
perceptions. 



^^This concept of "fairness" is derived in large part from Grace 
Carter's A Review of the literature Concerning the NSF Peer 
Review System , prepared for the Division of Policy Research and 
Analysis, National Science Foundation, July 18, 1986, p. 3. 
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Surveys 



As described in section 2 of this fact sheets over 60 
percent of the scientists surveyed by a 1986 Sigma Xi survey 
believed that receiving federal research funds was based on "who 
you know" and knowledge of prior research work by the granting 
agency. (However^ over 40 percent of the respondents were from 
industry and government and their perspectives were not reported 
separately from university scientists^ nor is it clear how much 
experience with the academic research award process they had 
had . ) 

Twenty-Height percent of the scientists who had served on NIH 
public advisory groups surveyed in a 1976 NIH peer review study 
believed the NIH system to embody a moderate or higher degree of 
cronyism, 27 (The frequency of this behavior by participants in 
the NIH process was not noted ^ nor has GAO independently 
evaluated whether subsequent changes in the NIH selection of 
reviewers or procedures related to peer review have diminished 
the occurrence of favoritism entering into award decisions.) 

Another survey of a sample of about 300 applicants for 
research support from NIH's National Cancer Institute in 1980 and 
198128 showed that approximately 40 percent agreed or strongly 
agreed that the initial review groups were controlled by "old boy 
networks"; a little over 25 percent were undecided on this. 
Similarly^ 35 percent agreed or strongly agreed that reviewers 
are biased against researchers in nonmajor universities or 
institutions in certain regions of the United States; 34 percent 
of the respondents were undecided on this issue. In comparison^ 
17 percent and 5 percent either agreed or strongly agreed that 
reviewers are biased against young researchers ^ or against women 



27 survey of the NIH Grant Peer Review System. Analysis of Survey 
Responses and Comments ^ U^S. Department of Healthy Education/ and 
Welfare^ National Institutes of Healthy July 1^ 1977^ p. 1-^26* 



^SThe survey was performed by Gillespie ^ Chubin^ and Kurzon and 
published as "Experience with NIH Peer Reviews Researchers' 
Cynicism and Desire for Change" in Science^ Technology ^ and Human 
Values 1. Vol. 10^ Issue 3, Summer 1985,^ ppr 44-54. In 
interpreting their views on NIH peer review^ it is important to 
note the percentage of respondents that characterized their 
previous success in obtaining research funding asi 62 percent 
successful or highly successful (more proposals funded than 
declined)^ 22 percent mixed success (no more than half of 
previous proposals funded)^ and 16 percent unsuccessful (no prior 
proposals funded). 
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and minorities r respectively. A survey by NSP of the views of 
the peer review process by all the researchers who applied for 
funds from NSP during fiscal year 1985 (whether daclined or 
awarded) will be completed early in 1987, 

Analyses of NSF award decisions 

Analyses of NSF award decisions during the past 10 years 
showed that I 

— There is no conclusiye statistical evidence that 
reviewers for NSF are biasing their reviews of proposals 
based on the applicant's age^ or gender^ or the 
geographic location or prestige of the applicant's 
institution* 29 (There were no data on racial background 
of the applicants,) 

" Peer reviewers give substantially better ratings to 

renewal applications than to first-time applications^ and 
women and young investigators constitute a smaller 
proportion of renewal applications than of first-time 
applications. The degree of higher ratings on renewals 
may be appropriate^ since they were judged of high 
quality when first awarded. 

— The ratings provided by the peer reviewers are the most 
important influence on the program officer's award 
recommendation* 

— Evidence is mijced on whether the amount of research done 
by an applicant's institution has an effect on the NSP 
program officer's recommendation^ with two stuaies 
offering opposing conclusions, 

" There is no firm answer on whether geographic region 
independently affects program officer's decisions, 

RELIABILITY AND EFFECTIVENBSS 

Reliability refers to the role of chance in determining the 
outcome of the NSF award decision process and focuses on the 
degree to which there is consensus regarding the scientific merit 



^^This summary of statistical evidence is based in large part on 
" A Review of the Literature Concerning the NSF Peer Review 
System " by Grace Carter, prepared forthe division of Policy 
Research and Analysis , NSF^ July 1986, We also rely on the 
findings of the report by Cole, et. al*, performed for the 
National Academy of Sciences and the ti "fairness studies" which 
were performed by the NSP evaluation staff (listed in app. I), 
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of individual proposals. Effectiveness refers to the outcome of 
projects selected for funding (i.e.^ Did they promote the "best" 
science?). There have been few studies f the reliability and 
effectiveness of peer review. Two studies performed on past NSF 
funding decisions^ on actions by NSP program officers during 
fiscal year 1976 and fiscal year 198U respectively^ found that 
reviewers disagree on the quality of proposals across the entire 
range of proposals, in that a new set of reviewers more than a 
year later would have reversed about a quarter of the original 
NSF funding decisions, 30 Thus, some have argued that chance can 
play a significant role in whether a proposal gets funded since 
different group of reviewers may come to an alternate conclusion 

As for the effectiveness of peer review, studies have shown 
the share of significant advances in a particular field of 
science, for example in chemistry, that have been supported by 
NSF, We are not aware, however, of studies that have compared 
the quality of research supported by NSF as a result of peer 
review of proposals and the research supported by other 
government or private organizations that do not rely on peer 
review in the same way. 



Ibid , p. 17. The Carter review cites the 1981 Cole study and 
the NSF Evaluation Staff Study 82-^1 (see Appendix 1). 
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SELECTED BIBLIOGRAPHY OP STUDIES OF PEER REVIEW 



The Administration o£ Government-Supported Research at 
UniverBitiee (the "Westrate report")^ 1966, 

Biomedical Science and its Administration ; A Study of The 
National Institutes of Healthy Report to the President (the 
"Wooldridge report 19657 " " " 

Carter^ G,M. Peer Review^ Citations y and Biomedical Research 
Policy: NIH Grants to MedrcaT School Faculty . Department of 
Healthy Education, and Welfare/the Rand Corporation , R-- 1 583^ 
HEW, Dec. 1974* 

Carter, G*M. A Review of the Literature Concerning the NSF Peer 
Review System . Prepared for the Division of Policy Research and 
Analysis 7 National Science Foundation, July 1986, 

Carter, G.M. What We Know and Do Not Know About the NIH Peer 
Review System t National Institutes of Eealth/the Rand 
Corporation, N-1 878-RC/NIH, June 1982. 

Cole, J.R., and Cole, Peer Review in the National Science 

Foundation , Phase 2. Washington, D.C.s Nat ionir Academy Press, 
1981 . 

Fairness of the NSF Award Decision Process , Fiscal Year 1982. 
Evaluation Staff Stuuies 84-^1, National Science Foundation, 
Jan. 1984. 

Fairness of the NSF Award Decision Process , Fiscal Year 1983. 
Evaluation Staff Studies 85^1, National Science Foundation, 
Mar. 1985. 

Final Report. NSF Advisory Committee on Merit Review . National 
Science Foundation, September 25, 19867 

Geographic and Institutional Effect on the Award Decision , 
Evaluation Staff Studies 81-^2. National Science" Foundation, 
Nov. 1981. 

Grants Peer Reviews Opinions on the NIH Grants Peer Review 

System , Phase ll of the Report to the Director, NIH. National 
Institutes of Health, Dec. 1978, 
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'^"IuL!L!f!f^" °5 Nationa l Institutes of H..n^h Committee on 
interstate ana Foreign Conimerce, U.S. House of Representatives 
Washington, D.C.i U.S. Government Printing Officeriug" 1975 ' 

National Science Found ation Peer Review; Specia l Oversight 
H||rin£S. subcommittee on Science, Research, and Technology, 
Committee on Science and Technology, U.S. House of """^''yy' 
olfilefms!^^* D.C./U.S. Government Printing 

^ xf (preliminary report). New Haven, Conn.. 

Sigma Xi, The Scientific Research Society, 1986. 

The NSF Post-Performanc e Evaluation study . Evaluation Staff 
fatudies 84-2. National Science Foundation, 1984. 

Steelman, John R. The Nation's Medical Research . (Volume 5 of 
science and Public Policy , a r..... .h. . ^1°^^"^^ ^ ^gf 

Report of the Presiden t's Biomedical Research Panel (established 
pursuant to Public Law 93-352.), 1 976. — — 

"^E'f Ji!!"^,.^^"';^ ^7'^ Advisory com m ittee on the Management of 

NIH Research contracts and Grants (the "Rnin^ .^p^.^-..^^ „ g 

Department ot Health, Education, and Welfare, 1966. 

Reviewer and Proposer Similarity and its Effect o n Award 

p||iSion|,^Evaluation staff studies ;6-1. National Science 

^^u:^r Assessing the Quality of Research Prn pn^.i. 

Vtio'"^ ' ^^tional Commislion on Research, May 

A Study of the Fairnes s of the NSF Award Decision P ron^sc. 

Aug ^982? ^^"^'^^ Nation al Science Founda tion , 
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APPENDIX II APPENDIX II 

RESOURCES USED BY NSF WHEN RECRUITING PEER REVIEWERS 



o Computer lEed lists or files of potential reviewers, 

o Lists o£ Buggested reviewers sent by the applicants 
themselves , 

o Authors of publications containing related research cited 
as part of the proposal. 

o Current literature in the area of the proposal* 

o Other reviewers who have served NSP. 

o Program officer's knowledge of other researchers in the 
field* 

o Contacts made by NSP staff at professional meetings* 

o Speeches and visits made in connection with NSP "outreach" 
efforts to undergraduate institutions ^ minority 
institutions # and other types of institutions* 



Source: NSF officials* 
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APPENDIX III APPENDIX 111 

NSF CRITERIA FOR SELECTING PEER REVIEWERS 



o Peer reviewers should have 

speaial knowledge of the science and engineering 
fields^ 

— broader or more generalized knowledge of the science 
and engineering fields^ and 

broad knowledge of the infrastructure of the science 
and engineering enterprise and its educational 
activities . 



o To the extent possible^ reviewers should reflect a balance 
among various characteristics such as geography^ type of 
institution^ and underrepresented groups. 

o Reviewers should be chosen in accordance with NSF conf lict--of- 
interest rules and standards of conduct to avoid bias in the 
review of proposals. 

o Too frequent use of any one reviewer or institution must be 
avoided, 

o Additional criteria for selecting reviewers who serve on 
panels t 

— Age distribution. Members should be selected from as 
broad a range of age groups as is feasible. 

— Regular rotation^ Members can serve a maximum of 3 
years. 

Academic and nonacademic impact . Members should 
represen^dif f erent size^ as well as public and private 
institutions, whenever possible^ concurrent or 
successive appointments of individuals from the same 
institution should be avoided. Representatives from 
outside the academic community are also desirable in 
some instances . 

— Public Impact. Where pertinent^ some members should be 
representative of regions^ organizations^ or segments of 
the public directly affected by issues under 
consideration . 

Sources NSF Proposal and Award Manual . 
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RESOURCES USED BY NIH WHEN RECRUITING PEER REVIEWERS 
FOR INITIAL PEER REVIEW COMMIWEES 

o Knowledge of the executive secretary of the committee of 
the field and the scientists who work within it, 

o Executive secretary's contacts at scientific meetings* 

o Institute staff recommendations . 

o Research grant applications and awards. 

o Research publications. 

o Recommendations of the members of the initial peer review 
committees * 



Source: NIH officials. 
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NIH CRITERIA FOR SELECTING 
PEER REVIEWERS 



o Peer reviewers must have demonstrated competence and 
achievement as independent investigators . 

NIH peer reviewers must meet the following requirementsi 

— No more than one member per institution, 

— The goal of membership is at least 17 percent ethnic 
minority and 23 percent female. 

No more than 15 percent membership may be from each of 
four states: New York^ California, Massachusetts , and 
Texas* Section membership is suggested to be no more than 
10 percent from all other states individually . 

— NO more than 50 percent membership may be from any one of 
four geographic regions? South, Central, East, and West. a 



— No more than one federal employee except in unusual 
circumstances 

— A year must lapse before a committee member can be 
reappointed to serve on the same or any other committee. 

— No excessive service, as defined by an individual serving 
more than a total of 8 years in the past 12 years. 

After a member completes the term, a year must elapse 
before a member can be appointed from the same 
institution.^ 



Sources NIH officials and NIH written documents. 

^Only a requirement for serving on initial review groups (study 
sections) with NlH's Division of Research Grants. 
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RESPONSIBILITIES OF NIH EXECUTIVE SECRETARIES 
AND INSTITUTE STAFF IN AWARD SELECT ION ' 



O NIH EXECUTIVE SECRETARIES 

Identify and nominate the peer reviewers to serve on 
panels r orient new members. 

— Prepare agenda of the study section meeting. 

— Identify proposals that are special or incomplete and 
obtain additional information , as needed . 

Assign proposals to primary and secondary reviewers and 
ensure against conflict of interest. 

— Administer the peer panel meetings to ensure they are 
conducted in accordance with NIH policy and procedures* 

— * Write the summary statements that are the official agency 
record of the evaluations and recommendations made by peer 
reviewers and contain ^ among other things^ the review 
group's vote^ priority scores^ and summaries of their 
critique . 

— - Administer the site visits by peer reviewers on larger 
projects^ if necessary. 

O INSTITUTE STAFF 

— Nominate council members for vacancies on the advisory 
council and orient new members, 

— Identify applications whose priority score would not be in 
the fundable range ^ but whose program relevance^ 
uniqueness^ or potential impact merit recommending 
funding. 

— Prepare advisory council briefing books that contain the 
initial review group summary statements^ and indicate 
those proposals marked for special consideration. 

— Recommend to the Director which applications that have 
been approved by the Council should be funded. 

Sources NIH officials and documents. 
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mF StgPOiCT FOR UMDERRm^^EMm) GROtgS Pm IIBTITOTIOIB 



APP^IX VII 



Mecdianism 



Program title 



Goal 



FY 1986 
funding 
(millions) 



MINORITY 
II^STIGATO^ 
mo IieTITUTIONS 

Targeted 
programs 



Separately 

budgeted 

progran^ 



Addition^ 

research 

program 

St 



Minority ^search 
Initiation {mi) 



Minority Research 
Centers of Excellenoe 
(MRCE) 



tesearch in 
in Minority 
Institutions (RIMI) 



Minority 
Investigators 



Minority Institutions 
(includes 

Historically Black 
Colleges) 



To provide support to allow 
researchers have never 
received federal research 
sijpport as faculty members to 
establish ii^ependent research 
projects 

To increase the minority 
presence in scienoe and 
engineerir^ by ypgrading the 
research c^abllitiea of the 
strongest and most productive 
minority institutions, ^few in 
m 1987, Estimated supports $4 
million 

To help institutions with 
sizzle minority enrollments 
increase their research and 
research-related training 
c^abilities 

Sui^rt for Minority 
Investigators through the 
regular competitive research 
progrM^ 



$2.6 



5.0 



6.6 



12.8 



Total su^ort to 
minority 
investigators 
institutions 



$27,0 
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UNDEE^ADDATE 
IlKTITOTIONS 

Targeted 
programs 



Program title 



Regular progran 
support 



Rasaarch in 
Undergraduate 
Institutions (RUI) 



Goal 



Support for research and 
instrimentation proposals 
submitted directly to the 
regular disciplinary programs 

To strengthen the reseiircn 
environment in undergraduate 
academic departments within 
predominantly undergr^uate 
(nondoctoralj institutions 



FY 1986 
funding 
(millions) 



$37.5 



10*7 



Research Provide o^ortunities for 

C^portunities Arard faculty at undergrMuate 
(RQA) institutions to participate in 

research under aegis of KBP 
investigators at major research 
institutions 



Tbtal support to 
predominantly 
undergraduate 
institutions 
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Itechanism 



WDNEN 



Program title 



py 1986 
fundir^ 
(milliona) 



Targeted 
prog rains 



Research 

^portunities for 
Ptomen (ROT) 



To increase thit narters oJf 
women as full participants in 
the mainstream of the nation's 
research enterprise through 
research Initiation planning 
and career advanceirent grants 



$2.0 



Separately 

budgeted 

programs 



Vieiting 

Professorship for 
Women (VPW) 



To enhance TOmen's participa- 
tion in science and engineering 
by enabling women scientists 
and engineers experienced in 
independent research to serve 
as visiting professors 



2 0 



Mditional 
research 

program support 



Women Investigators 



Si^port for TOiren investigators 
through the regular cor^titive 
research programs 



63.5 



Total research 
support to women 
investigators 



Source I mP officials and internal documents. 
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APPENDIX VIII 



NIH SUPPORT FOR UKDmREPRESISNTED GROUPS AND INSTITOTIONS 



Prog ran 

ty^nority Biomedical 
Research Support 
Prog ran 



Research Centers in 
Minority Institutions 



Minority Access to 
Research Career 



Academic Research 
Enhancement Progran 



^al 

Promote ndnority fac^ty 
and student participation 
in biomedical research and 
strengthen biomedical 
research capabilities at 
eligible institutions 

Enhance capacity of 
doctoral degree^ranting 
minority institutions to 
conduct biomedical and/or 
behavioral research by 
strengthening tJteir 
research environment 

Address the research 
training o£ ethnic 
minorities in the 
biomedical sciences 

Stimulate research in 
smaller^ less ptominent 
colleges and universities 
that provide undergraduate 
and graduate training for 
a significant number of 
our nation's research 
scientists but that have 
not shared adequately in 
tile growth of OTH 
extramural arards 



m. of 

amrds 

107 



16 



81 



184 



FY 1986 
funding 
( millions ) 

$32.3 



9.8 



7,2 



12,1 
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Program 

Minority High School 
Stirfent Rasearch 
l^prentice Program 



Goal 

Ito provide meaningful 
experience in various 
^pects of health'-related 
research in the 
eKpectation that some 
apprentices will decide to 
pursue careers in 
bioiredical research 



^.of 
299 



FY 1986 
funding 
( millions ) 

1.5 



Minority Biomedical 
Research S^pDrt 
Hiematic Project 
Grants 



Minority Biomedicri 
Research Support 
Grant ProgrMi for 
Unde rg r aduat e 
Colleges 



To promote increased 
faculty and 
interdepartinental 
collaboration "Uirough 
programs toat focus on 
specific research themes 
or scientific disciplines 
at davaloping minority 
institutions . These 
gr^ts are intended to 
strengthen the biomedical 
research c^ability in 
defined areas aM to 
attract other cortpetent 
biomedic^ scientists 
through an in^roved 
research environment. 

To enrich lAa rasearch 
environment at eligible 
undergrMuate institutions 
with the purposes of 
motivating students toward 
biomedical research 
careers and providir^ 
faculty with opportimities 
to partici^te in 
bionedical rasearch 



10 



Total FY "86 su^rt 
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NIH Suppoirt for Minorities and Women 
on Competing Individual Research ProjectSy FY 1986 ^ 



Estimated support b 
(in thousands) 



Total 

Percent of total 



$919,470 
100*0 



Minorities 
Percent o£ total 



185,51 1 
9,3 



Women 

Percent of total 



¥122,290 
13.3 



Women or minorities 
Percent of total 



$199,525 c 
21 .7 




fiscal year 1986 (i.e., new awards, competitive renewals, and 
competing supplements), 

^Estimates are made for nonresponses * 

^Because there is some overlap between the "women" and 
"minorities" categories, these categories don't add up to "women 
or minorities." 



Sourcei NIH officials. 
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COMPARISON OF NSF AND NIH PROJECT SELECTION SYSTEMS 



Proposal selection procedures NSF NIH 



Use of peers to determine scientific merit Y 

Dominant peer review method (s): 

Panel -^b y 

Ad hoc mail y n 

Ad hoc mail and panel y u 

Funding recommendatione made bys 

Peer reviewers N N 

In-house officials y y 

Funding decisions made by: 

Peer reviewers ig N 

In-house staff y y 

Selection of peer reviewers 

Written criteria for selection y y 

Criteria emphasize scientific expertise y y 
Criteria state numerical goals/requirements for* 

Geographic balance N Y 

Institutional balance N Y 

Women and minorities N Y 

Length of service y y 

Role of peer reviewers 

Provide scientific and technical review Y y 

Evaluate program priorities/relevance N Y 

Recommend proposal budget and duration N Y 

Recommend approval of proposal for award N Y 

Numerically rate proposal y y 

Make funding decisions N N 

Role of agency officials 

Select peer reviewers y y 

Evaluate proposals for scientific merit Y n 

Recommend proposal for funding y y 

Make funding decision y y 



ay=yes . 
^N^no. 
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Research infrastructure 

Agency considers the following when funding projects 
Geographic balance 

Targets 
Underrepresented groups 

Targets 

Less research-intensive institutions 
Targets 

Innovative^ high-risk proposals 

Targets 
Young and new researchers 

Targets 

Oversight and monitoring 

Agencies routinely provide the following to ensure 
system's fairness and openness 2 

Central computer file of peer attributes 

Regular rotation of peers 

Conf lict--of-interest rules 

Peer evaluations sent to applicant 

Appeals process for applicants 

External oversight 

Internal audit/evaluation 

Centrally monitored targets 

Source: NIH and NSP agency officials and documents. 
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